Skip to content

chore: remove retired sequential and staggered TTI SVGs - #304

Merged
dtice25 merged 2 commits into
masterfrom
devin/1786484713-retire-tti-svgs
Aug 11, 2026
Merged

dtice25 merged 2 commits into
masterfrom
devin/1786484713-retire-tti-svgs

Conversation

@dtice25

@dtice25 dtice25 commented Aug 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Sequential and staggered sandbox TTI tests were retired in #303, but their auto-generated sequential_tti.svg and staggered_tti.svg leaderboards still lived in the repo root and were still rendered in the README. This PR removes those SVGs, notes the retirement in README.md and METHODOLOGY.md, and updates benchmarks/sandbox/generate-svg.ts so the daily workflow no longer regenerates the retired charts.

Changes

  • Delete sequential_tti.svg and staggered_tti.svg
  • Remove the "Sequential TTI" and "Staggered TTI" image sections from README.md
  • Add a README callout that the sequential/staggered tests are retired and that historic JSON results remain in results/
  • Update METHODOLOGY.md to stop listing the deleted SVGs and to reference the archived result directories
  • Fix benchmarks/sandbox/generate-svg.ts default mode list to ['burst'] so the daily run does not regenerate retired sequential/staggered SVGs

Link to Devin session: https://app.devin.ai/sessions/7d84ba08f97d48e5b0549c0508e320c1
Requested by: @dtice25


Open in Devin Review

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…ent in README and METHODOLOGY

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@dtice25 dtice25 self-assigned this Aug 11, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@open-cla

open-cla Bot commented Aug 11, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

View 2 additional findings in Devin Review.

Open in Devin Review

Comment thread METHODOLOGY.md Resolved
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@dtice25
dtice25 merged commit b224f11 into master Aug 11, 2026
6 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Burst

# Provider Score Median TTI P95 P99 Status
1 isorun 98.8 0.08s 0.17s 0.17s 10/10
2 arker 98.4 0.15s 0.17s 0.17s 10/10
3 createos 98.3 0.10s 0.29s 0.29s 10/10
4 lightning 97.9 0.18s 0.25s 0.25s 10/10
5 northflank 97.9 0.18s 0.26s 0.26s 10/10
6 archil 97.7 0.22s 0.24s 0.24s 10/10
7 blaxel 96.4 0.35s 0.37s 0.37s 10/10
8 declaw 96.3 0.29s 0.50s 0.50s 10/10
9 opencomputer 96.1 0.35s 0.43s 0.43s 10/10
10 upstash 93.6 0.62s 0.66s 0.66s 10/10
11 vercel 92.2 0.47s 1.25s 1.25s 10/10
12 superserve 91.2 0.49s 1.46s 1.46s 10/10
13 runloop 91.0 0.88s 0.94s 0.94s 10/10
14 modal 90.5 0.67s 1.35s 1.35s 10/10
15 mosaic 90.5 0.88s 1.06s 1.06s 10/10
16 e2b 90.2 0.95s 1.03s 1.03s 10/10
17 beam 89.8 1.02s 1.03s 1.03s 10/10
18 cloud-run 84.5 1.39s 1.80s 1.80s 10/10
19 tensorlake 83.2 1.60s 1.81s 1.81s 10/10
20 sandbox0 82.4 1.67s 1.89s 1.89s 10/10
21 cloudflare 80.6 1.84s 2.09s 2.09s 10/10
22 sail 78.7 2.13s 2.14s 2.14s 10/10
23 daytona 48.2 1.07s 22.55s 22.55s 9/10
24 run-cloud 15.0 8.28s 8.83s 8.83s 10/10
25 codesandbox 0.0 10.86s 11.57s 11.57s 10/10
26 hopx 0.0 0.00s 0.00s 0.00s 0/10
27 tenki 0.0 0.00s 0.00s 0.00s 0/10

View full run · SVGs available as build artifacts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant